Modeling User Feedback Using a Hierarchical Graphical Model for Interactive Image Retrieval

Relevance feedback is an important mechanism for narrowing the semantic gap in content-based image retrieval and the process involves the user labeling positive and negative images. Very often, it is some specific objects or regions in the positive feedback images that the user is really interested in rather than the entire image. This paper presents a hierarchical graphical model for automatically extracting objects and regions that the user is interested in from the positive images which in turn are used to derive features that better reflect the user’s feedback intentions for improving interactive image retrieval. The novel hierarchical graphical model embeds image formation prior, user intention prior and statistical prior in its edges and uses a max-flow/min-cut method to simultaneously segment all positive feedback images into user interested and user uninterested regions. An important innovation of the graphical model is the introduction of a layer of visual appearance prototype nodes to incorporate user intention and form bridges linking similar objects in different images. This architecture not only makes it possible to use all feedback images to obtain more robust user intention prior thus improving the object segmentation results and in turn enhancing the retrieval performance, but also greatly reduces the complexity of the graph and the computational cost. Experimental results are presented to demonstrate the effectiveness of the new